Generalized Hopfield networks for constrained optimization

A twofold generalization of the classical continuous Hopfield neural network for modelling constrained optimization problems is proposed. On the one hand, non-quadratic cost functions are admitted corresponding to non-linear output summation functions in the neurons. On the other hand it is shown under which conditions various (new) types of constraints can be incorporated directly. The stability properties of several relaxation schemes are shown. If a direct incorporation of the constraints appears to be impossible, the Hopfield-Lagrange model can be applied, the stability properties of which are analyzed as well. Another good way to deal with constraints is by means of dynamic penalty terms, using mean field annealing in order to end up in a feasible solution. A famous example in this context is the elastic net, although it seems impossible - contrary to what is suggested in the literature - to derive the architecture of this network from a constrained Hopfield model. Furthermore, a non-equidistant elastic net is proposed and its stability properties are compared to those of the classical elastic network. In addition to certain simulation results as known from the literature, most theoretical statements of this paper are validated with simulations of toy problems while in some cases, more sophisticated combinatorial optimization problems have been tried as well. In the final section, we discuss the possibilities of applying the various models in the area of constrained optimization. It is also demonstrated how the new ideas as inspired by the analysis of generalized continuous Hopfield models, can be transferred to discrete stochastic Hopfield models. By doing so, simulating annealing can be exploited in other to improve the quality of solutions. The transfer also opens new avenues for continued theoretical research

An analysis of various elastic net algorithms

The Elastic Net Algorithm (ENA) for solving the Traveling Salesman Problem is analyzed applying statistical mechanics. Using some general properties of the free energy function of stochastic Hopfield Neural Networks, we argue why Simic's derivation of the ENA from a Hopfield network is incorrect. However, like the Hopfield-Lagrange method, the ENA may be considered a specific dynamic penalty method, where, in this case, the weights of the various penalty terms decrease during execution of the algorithm. This view on the ENA corresponds to the view resulting from the theory on `deformable templates', where the term stochastic penalty method seems to be most appropriate. Next, the ENA is analyzed both on the level of the energy function as well as on the level of the motion equations. It will be proven and shown experimentally, why a non-feasible solution is sometimes found. It can be caused either by a too rapid lowering of the temperature parameter (which is avoidable), or by a peculiar property of the algorithm,namely, that of adhering to equidistance of the elastic net points. Thereupon, an alternative, Non-equidistant Elastic Net Algorithm (NENA) is presented and analyzed. It has a correct distance measure and it is hoped to guarantee feasibility in a more natural way. For small problem instances, this conjecture is confirmed experimentally. However, trying larger problem instances, the pictures changes: our experimental results show that the elastic net points appear to become `lumpy' which may cause non-feasibility again. Moreover, in cases both algorithms yield a feasible solution, the quality of the solution found by the NENA is often slightly worse than the one found by the original ENA. This motivated us to try an Hybrid Elastic Net Algorithm (HENA), which starts using the ENA and, after having found a good approximate solution, switches to the NENA in order to guarantee feasibility too. In practice, the ENA and HENA perform more or less the same. Up till now, we did not find parameters of the HENA, which invariably guarantee the desired feasibility of solutions

Internetbeveiliging: een beheerperspektief

Binnen het generieke kader van het beheer van informatiesystemen wordt de beveiligingsproblematiek rond het gebruik van het Internet onder de loep genomen en geanalyseerd. De invalshoek is zowel technisch als organisatorisch. Na een korte analyse van het nut van computernetwerken voor een organisatie worden - aan de hand van een basaal communicatiemodel - de potentiele risico's van Internetgebruik in kaart gebracht. Op basis hiervan kan een (voor de specifieke organisatie) noodzakelijk beveiligingsnivo worden gedefinieerd. Na vaststelling hiervan dienen bijpassende technische en organisatorische maatregelen te worden genomen. Deze (noodgedwongen zeer dynamische) aanpak wordt geformaliseerd met behulp van een struktuurmodel. In aparte kaders worden zowel de onderscheiden typen risico's als de onderscheiden soorten maatregelen nader toegelicht aan de hand van allerlei praktijkvoorbeelden

Associative conceptual space-based information retrieval systems

In this `Information Era' with the availability of large collections of books, articles, journals, CD-ROMs, video films and so on, there exists an increasing need for intelligent information retrieval systems that enable users to find the information desired easily. Many attempts have been made to construct such retrieval systems, including the electronic ones used in libraries and including the search engines for the World Wide Web. In many cases, however, the so-called `precision' and `recall' of these systems leave much to be desired. In this paper, a new AI-based retrieval system is proposed, inspired by, among other things, the WEBSOM-algorithm. However, contrary to that approach where domain knowledge is extracted from the full text of all books, we propose a system where certain specific meta-information is automatically assembled using only the index of every document. This knowledge extraction process results into a new type of concept space, the so-called Associative Conceptual Space where the `concepts' as found in all documents are clustered using a Hebbian-type of learning algorithm. Then, each document can be characterised by comparing the concepts as occurring in it to those present in the associative conceptual space. Applying these characterisations, all documents can be clustered such that semantically similar documents lie close together on a Self-Organising Map. This map can easily be inspected by its user

Constrained optimization with a continuous Hopfield-Lagrange model

In this paper, a generalized Hopfield model with continuous neurons using Lagrange multipliers, originally introduced Wacholder, Han &Mann [1989], is thoroughly analysed. We have termed the model the Hopfield-Lagrange model. It can be used to resolve constrained optimization problems. In the theoretical part, we present a simple explanation of a fundamental energy term of the continuous Hopfield model. This term has caused some confusion as reported in Takefuji [1992]. It led to some misinterpretations which will be corrected. Next, a new Lyapunov function is derived which, under some dynamical conditions, guarantees stability of the the system. We explain why a certain type of frequently used quadratic constraints can degenerate the Hopfield-Lagrange model to a penalty method. Furthermore, a difficulty is described which may arise if the method is applied to problems with `hard constraints'. The theoretical results suggest a method of using the Hopfield- Lagrange model. This method is described and applied to several problems like Weighted Matching, Crossbar Switch Scheduling and the Travelling Salesman Problem. The relevant theoretical results are applied and compared to the computational ones. Various formulations of the constraints are tried, of which one is a new approach, where a multiplier is used for every single constraint

On the statistical mechanics of (un)constrained stochastic Hopfield and 'elastic' neural networks

Stochastic binary Hopfield models are viewed from the angle of statistical mechanics. After an analysis of the unconstrained model using mean field theory, a similar investigation is applied to a constrained model yielding comparable general explicit formulas of the free energy. Conditions are given for which some of the free energy expressions are Lyapunov functions of the corresponding differential equations. Both stochastic models appear to coincide with a specific continuous model. Physically, the models are related to spin and Potts glass models. Also, a `complementary' free energy function of both the unconstrained and the constrained model is derived. The analysis culminates in a very general framework for analyzing constrained and unconstrained Hopfield neural networks: the stationary points of the corresponding free energy appears to coincide exactly with the set of equilibrium conditions of the corresponding continuous Hopfield neural network. Moreover, the relationship with `elastic net' algorithms is analyzed: it is proved that this class of algorithms cannot be derived from the theory of statistical mechanics (as sometimes is supposed), but should be considered as a special `penalty method', namely as one with dynamical penalty weights. We mention some experimental results and discuss implications for the use of the various models in resolving constrained optimization problems

Competitive exception learning using fuzzy frequency distributions

A competitive exception learning algorithm for finding a non-linear mapping is proposed which puts the emphasis on the discovery of the important exceptions rather than the main rules. To do so,we first cluster the output space using a competitive fuzzy clustering algorithm and derive a fuzzy frequency distribution describing the general, average system's output behavior. Next, we look for a fuzzy partitioning of the input space in such away that the corresponding fuzzy output frequency distributions `deviate at most' from the average one as found in the first step. In this way, the most important `exceptional regions' in the input-output relation are determined. Using the joint input-output fuzzy frequency distributions, the complete input-output function as extracted from the data, can be expressed mathematically. In addition, the exceptions encountered can be collected and described as a set of fuzzy if-then-else-rules. Besides presenting a theoretical description of the new exception learning algorithm, we report on the outcomes of certain practical simulations

Financial Markets Analysis by Probabilistic Fuzzy Modelling

For successful trading in financial markets, it is important to develop financial models where one can identify different states of the market for modifying one???s actions. In this paper, we propose to use probabilistic fuzzy systems for this purpose. We concentrate on Takagi???Sugeno (TS) probabilistic fuzzy systems that combine interpretability of fuzzy systems with the statistical properties of probabilistic systems. We start by recapitulating the general architecture of TS probabilistic fuzzy rule-based systems and summarize the corresponding reasoning schemes. We mention how probabilities can be estimated from a given data set and how a probability distribution can be approximated by a fuzzy histogram. We apply our methodology for financial time series analysis and demonstrate how a probabilistic TS fuzzy system can be identified, assuming that a linguistic term set is given. We illustrate the interpretability of such a system by inspecting the rule bases of our models

Relative Distress and Return Distribution Characteristics of Japanese Stocks, a Fuzzy-Probabilistic Approach

In this article, we demonstrate that a direct relation exists between the context of Japanese firms indicating relative distress and conditional return distribution properties. We map cross-sectional vectors with company characteristics on vectors with return feature vectors, using a fuzzy identification technique called Competitive Exception Learning Algorithm (CELA)1. In this study we use company characteristics that follow from capital structure theory and we relate the recognized conditional return properties to this theory. Using the rules identified by this mapping procedure this approach enables us to make conditional predictions regarding the probability of a stock's or a group of stocks' return series for different return distribution classes (actually return indices). Using these findings, one may construct conditional indices that may serve as benchmarks. These would be particularly useful for tracking and portfolio management

Probabilistic and Statistical Fuzzy Set Foundations of Competitive Exception Learning

Recently, a Competitive Exception Learning Algorithm (CELA) was introduced [1, 2]. This algorithm establishes an optimal mapping from a (continuous) M-dimensional input sample space to an N-dime